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November  5,  1992 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 

SUBJECT:  Audit  Report  on  the  Critical  Design  Review  Process 
for  Major  Defense  Acquisition  Programs 
(Report  No.  93-017) 


We  are  providing  this  final  audit  report  for  your  review  and 
comments.  On  June  30,  1992,  a  draft  of  this  report  was  provided 
to  you  for  comments.  As  of  October  28,  1992,  we  had  not  received 
responses  to  the  draft  report. 

DoD  Directive  7650.3  requires  that  all  audit  recommendations 
be  resolved  promptly.  Therefore,  comments  must  be  made  by 
January  04,  1993.  If  appropriate,  you  may  propose  alternative 
methods  for  accomplishing  desired  improvements.  This  report  does 
not  identify  any  quantifiable  monetary  benefits.  Recommendations 
are  subject  to  resolution  in  accordance  with  DoD  Directive  7650.3 
in  the  event  of  nonconcurrence  or  failure  to  comment. 

We  appreciate  the  courtesies  extended  to  our  audit  staff. 

If  you  have  any  questions  on  this  audit,  please  contact 
Mr.  Russell  A.  Rau,  Program  Director,  at  (703)  693-0186 
(DSN  223-0186)  or  Mr.  Brian  Flynn,  Project  Manager,  at 
(703)  693-0400  (DSN  223-0400).  Appendix  F  lists  the  planned 
distribution  of  this  report. 


Robertr  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 

Enclosure 

cc: 

Secretary  of  the  Army 
Secretary  of  the  Navy 
Secretary  of  the  Air  Force 
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Office  of  the  Inspector  General 

AUDIT  REPORT  KO.  93-017  November  5,  1992 

(Project  No.  1AE-0051) 

THE  CRITICAL  DESIGN  REVIEW  PROCESS 
FOR  MAJOR  DEFENSE  ACQUISITION  PROGRAMS 

EXECUTIVE  SUMMARY 

Introduction.  Critical  design  review  (CDR)  is  defined  as  a 
review  to  determine  if  the  detailed  system  design  meets  the 
performance  and  engineering  requirements  of  development 
specifications.  During  CDR,  the  Government  must  ensure  that  all 
design  areas  are  adequately  examined,  that  design  weaknesses  are 
identified,  and  that  solutions  for  design-related  problems  are 
available.  The  Government  must  use  the  results  of  the  CDR  to 
assess  the  readiness  of  the  system  to  progress  to  the  next 
acquisition  phase.  The  design  reviews  and  associated  testing  of 
design  features  let  the  Government  review  the  complete  system 
design  and  evaluate  its  capability  to  satisfy  total  mission 
requirements.  Presently,  DoD  proposes  acquisition  strategies 
that  focus  on  developing,  but  not  on  producing,  systems.  Those 
strategies  require  that  CDR  be  effective  and  not  rely  on  later 
production  efforts  to  resolve  design  deficiencies. 

Objectives.  The  overall  audit  objective  was  to  evaluate  the 
effectiveness  of  the  CDR  process  for  major  Defense  acquisition 
programs.  We  also  evaluated  the  internal  controls  for  the  CDR 
process. 

Audit  Results.  The  CDR  process  for  the  six  major  Defense 
acquisition  programs  reviewed  was  ineffective  in  ensuring  that 
the  detailed  design  and  supporting  documentation  satisfied 
mission  requirements  before  proceeding  with  production.  We 
identified  15  types  of  deficiencies  in  the  CDR  processes  of  the 
six  major  Defense  acquisition  programs.  We  discussed  those 
deficiencies  in  this  report  under  the  categories  of  planning, 
performance  and  control,  and  acceptance  of  the  CDR  process. 
Because  existing  directives,  guidance,  and  procedures  were 
inadequate,  the  primary  objectives  of  CDRs  have  not  been  met. 
Premature,  ineffective,  and  incomplete  CDRs  have  not  established 
that  major  Defense  acquisition  programs  can  meet  performance 
requirements.  Also,  those  CDRs  have  not  stabilized  design  for 
production  or  provided  early  identification  of  cost,  schedule, 
and  performance  deficiencies  for  the  system  design.  As  a  result, 
the  Government  has  paid  for  incomplete  design  efforts  and  did  not 
maintain  adequate  control  over  the  design  process. 
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Internal  Controls.  The  audit  identified  material  internal 
control  weaknesses.  Controls  to  track  actions  that  correct 
current  and  prior  design  deficiencies  were  inadequate  to  ensure 
that  corrective  actions  were  sufficient  or  actually  taken.  Also, 
controls  were  not  established  to  ensure  that  senior  acquisition 
management  received  enough  information  on  CDR  results  to  make 
sound  management  decisions.  Part  I  discusses  the  controls 
assessed. 

Potential  Benefits  of  Audit.  The  monetary  benefits  to  be 
realized  by  implementing  the  recommendations  were  not  readily 
quantifiable  (Appendix  D) . 

Summary  of  Recommendations.  We  recommended  that  the  Under 
Secretary  of  Defense  for  Acquisition: 

o  establish  CDR  as  an  acquisition  program  baseline  for 
Defense  programs; 

o  establish  criteria  to  start  and  complete  all  phases  of 

CDR; 


o  require  that  CDR  be  made  a  separately  priced  contract 
line  item  in  development  contracts  with  contractual  provisions 
for  individual  withholds  for  each  open  action  item  until 
corrective  action  is  completed; 

o  report  delays  in  completion  of,  and  significant  program 
changes  because  of,  CDRs  to  senior  acquisition  officials;  and 

o  expedite  issuance  of  revised  guidance  for  the  CDR 
process. 

Management  Comments.  No  comments  were  received  in  response  to 
the  draft  report.  Comments  to  the  final  report  are  requested 
from  the  Under  Secretary  of  Defense  for  Acquisition  by 
January  4,  1993.  We  were  informed  that  comments  were  being 
prepared  and  would  be  submitted  in  response  to  this  final  report. 
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The  Acquisition  Management  Directorate,  Office  of  the  Assistant 
Inspector  General  for  Auditing,  DoD,  prepared  this  report. 
Copies  of  the  report  can  be  obtained  from  the  Information 
Officer,  Audit  Planning  and  Technical  Support  Directorate,  at 
(703)  614-6303  (DSN)  224-6303). 


PART  I  -  INTRODUCTION 


Background 

Critical  design  review  (CDR)  is  essential  in  a  well-disciplined, 
full-scale  development  program.  DoD  Instruction  5000.2,  "Defense 
Acquisition  Management  Policies  and  Procedures,"  February  23, 
1991,  defines  CDR  as  a  review  conducted  to: 

o  determine  if  the  detailed  design  of  a  system  meets  the 
performance  and  engineering  requirements  of  the  development 
specification; 

o  establish  the  detailed  design  compatibility  among  the  end 
item  and  other  items  of  equipment,  facilities,  computer  programs, 
and  personnel; 

o  assess  producibility  and  risk  areas;  and 

o  review  the  preliminary  product  specifications. 

DoD  Instruction  5000.2  does  not  give  specific  guidance  to  conduct 
the  CDR  but  does  require  tailored  application  of  Military 
Standard  (Mil-Std)  152 IB,  "Technical  Reviews  and  Audits  for 
Systems,  Equipments,  and  Computer  Software,"  June  4,  1985. 

The  Government  must  identify  specific  design  reviews  to  impose  on 
a  development  program  and  include  them  in  Requests  for  Proposals, 
with  any  resulting  contracts.  Generally,  the  system-engineering 
management  process  includes  system,  preliminary,  and  critical 
design  reviews  phased  to  the  maturation  of  the  design.  The  CDR 
is  done  when  a  program  nears  the  end  of  its  design  phase,  during 
engineering  and  manufacturing  development.  The  purpose  of  the 
CDR  is  to  identify  design  weaknesses  or  faults. 

During  the  CDR,  the  Government  must  ensure  that  all  design  areas 
are  adequately  examined,  that  design  weaknesses  are  identified, 
and  that  solutions  for  design-related  issues  are  available.  The 
Government  must  use  the  CDR  results  to  assess  the  readiness  of 
the  system  to  progress  to  the  next  acquisition  phase.  If  the 
system  under  development  is  complex,  the  CDR  may  be  an 
incremental  review.  Separate  CDRs  are  performed  for  the  system 
computer's  hardware  and  software.  The  CDRs  and  associated 
testing  of  design  features  let  the  Government  review  the  complete 
system  design  and  evaluate  its  capability  to  satisfy  total 
mission  requirements.  The  CDR  process  also  lets  the  Government 
develop  the  production  configuration.  The  CDR  is  conducted  on 
each  configuration  item  before  design  release  and  production. 

Presently,  DoD  is  emphasizing  acquisition  strategies  that  focus 
on  developing,  but  not  on  producing,  Defense  systems. 
This  approach  necessitates  that  CDRs  be  effective  and  not  rely  on 
later  production  efforts  to  resolve  design  deficiencies. 


objectives 

Our  overall  objective  was  to  evaluate  the  effectiveness  of  the 
CDR  process  for  major  Defense  acquisition  programs.  We  also 
evaluated  the  internal  controls  for  the  CDR  process.  The  DoD 
Inspector  General  began  this  audit  to  determine  if  deficiencies 
identified  in  the  CDR  process  for  the  Navy  A-12  aircraft  are 
unique  to  the  A-12  Program  or  are  systemic  in  major  Defense 
acquisition  programs.  This  audit  is  one  of  a  series  to  assess 
the  overall  effectiveness  of  implementation  of  acquisition 
management  recommendations  incorporated  in  the  Secretary  of 
Defense's  "Defense  Management  Report." 

Scope 

To  accomplish  our  objectives,  we  examined  the  CDR  performed  on 
six  major  Defense  acquisition  programs  in  the  engineering  and 
manufacturing  development  phase.  Those  programs  were 

judgmentally  selected  and  included  the  following  two  programs 
each  from  the  Army,  Navy,  and  Air  Force: 

Army 

o  EPLRS  (Enhanced  Position  Location  Reporting  System) 
o  Javelin  (Advanced  Antitank  Weapon  System  -  Medium) 

Navy 

o  V-22  Osprey  (Joint  Advanced  Vertical  Lift  Aircraft) 
o  SSN-21  (Seawolf  Class  Nuclear  Attack  Submarine) 

Air  Force 

o  RSIP  (Radar  System  Improvement  Program  for  the 

Aircraft  Warning  and  Control  System  [AWACS]) 
o  Joint  STARS  (Joint  Surveillance  Target  and  Attack 

Radar  System) 

The  audit  universe  consisted  of  the  major  Defense  acquisition 
programs  listed  in  the  "Selected  Acquisition  Report." 

We  examined  the  design  and  design  review  requirements  in 

acquisition  plans,  acquisition  strategy  reports,  operational 
requirements  documents,  requests  for  proposals,  contracts, 
contract  modifications,  test  and  evaluation  master  plans, 
developmental  test  reports,  operational  test  and  evaluation 
reports,  minutes  of  design  review  meetings,  action  items  from 
design  reviews,  and  documentation  to  resolve  action  items.  We 
interviewed  the  program  management,  procurement  contracting 

officer.  Defense  plant  representative  office,  and  contractor 
personnel  to  determine  the  policies  and  procedures  followed  in 
design  review.  We  examined  the  CDR  information  given  to  senior 
acquisition  management  in  the  "Defense  Acquisition  Executive 
Summary"  and  other  reports. 
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This  program  results  audit  was  made  from  June  1991  through  Hay 
1992  and  included  the  CDR  records  and  actions  noted  above  from 
January  1982  to  May  1992.  The  audit  was  made  in  accordance  with 

auditing  standards  issued  by  the  Comptroller  General  of  the 

United  States  as  implemented  by  the  Inspector  General,  DoD,  and 
so  included  necessary  tests  of  internal  controls.  Appendix  E 
lists  activities  visited  or  contacted. 

Internal  Controls 

We  evaluated  the  adequacy  of  internal  controls  over  the  CDR 

process.  As  part  of  our  evaluation,  we  assessed: 

o  Federal,  DoD,  and  Component  regulations,  instructions, 
directives,  and  other  guidance  on  the  CDR  process; 

o  Defense  component  implementing  procedures  and  compliance 
with  regulations;  and 

o  oversight  of  the  CDR  process  by  the  Office  of  the 

Secretary  of  Defense  and  the  DoD  components. 

The  audit  identified  material  internal  control  weaknesses,  as 
defined  by  Public  Law  97-255,  Office  of  Management  and  Budget 
Circular  A-123,  and  DoD  Directive  5010.38.  Controls  to  track 
actions  that  correct  current  and  prior  design  deficiencies  were 
inadequate  to  ensure  that  corrective  actions  were  sufficient  or 
actually  taken.  Internal  controls  also  were  insufficient  to 
ensure  adequate  planning,  performance,  and  acceptance  of  CDRs. 
Recommendations  1.  through  4.,  if  implemented,  will  correct  these 
weaknesses.  Also,  controls  were  not  established  to  ensure  that 
senior  acquisition  management  received  enough  information  on  the 
CDR  results  to  make  sound  management  decisions.  Recommendations 
5.  and  6.,  if  implemented,  will  correct  these  weaknesses. 

This  report  will  be  provided  to  senior  officials  responsible  for 
internal  control  within  DoD  and  each  of  the  Military  Components 
(Appendix  F) . 

Prior  Audits  and  Other  Reviews 

We  identified  five  audits  and  one  Navy  special  inquiry  issued 
since  June  1989  or  currently  ongoing  that  identified  deficiencies 
in  the  CDR  process  for  major  Defense  acquisition  programs. 
Appendix  C  synopsis es  these  reports. 
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other  Matters  of  Interest 


The  Air  Force  commissioned  the  Air  Force  Systems  Command*  to 
study  the  preliminary  and  critical  design  review  processes  in  use 
for  its  major  Defense  acquisition  programs.  The  study  was  made 
from  February  1991  through  August  1991.  The  study  found  that 
significant  improvements  were  needed  to  strengthen  the 
Air  Force's  design-review  processes.  As  a  result,  a  draft  policy 
statement  and  guidelines  for  the  design  review  processes  have 
been  prepared.  This  explains  design  review  objectives, 
scheduling,  implementing  design  reviews,  participants' 
responsibilities,  and  later  actions.  Compliance  with  the  policy 
and  guidelines  will  be  required  for  all  acquisition  strategies 
forwarded  to  milestone  decision  authority  60  days  after  approval 
of  the  policy.  However,  the  policy  had  not  been  officially 
adopted  at  the  time  of  the  audit. 


On  July  1,  1992,  after  our  audit  work  was  completed,  the  Air 
Force  Systems  Command  and  the  Air  Force  Logistics  Command  merged 
to  become  the  Air  Force  Materiel  Command. 
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PART  II  -  FINDING  AND  RECOMMENDATIONS 


THE  CRITICAL  DESIGN  REVIEW  PROCESS 

The  CDR  process  for  major  Defense  acquisition  programs  was 
ineffective  in  ensuring  that  the  detailed  design  and  supporting 
documentation  satisfied  mission  requirements  before  proceeding 
with  production.  Deficiencies  in  the  planning,  performance, 
acceptance,  and  control  of  CDRs  were  identified.  Appendix  A 
summarizes  the  deficiencies  for  each  of  the  major  Defense 
acquisition  programs  included  in  this  audit.  The  process  was 
ineffective  due  to  inadequacies  in  and  ineffective  compliance 
with  existing  directives,  guidance,  and  procedures  for  planning, 
performance,  control,  and  acceptance  of  the  CDR  process.  As  a 
result,  the  primary  objectives  of  CDRs  were  not  met.  Premature, 
ineffective,  and  incomplete  CDRs  did  not  establish  that  major 
Defense  acquisition  programs  could  meet  performance  requirements. 
The  reviews  also  did  not  stabilize  design  for  production  or 
provide  early  identification  of  cost,  schedule,  and  performance 
deficiencies  for  the  system  design.  As  a  result,  the  Government 
paid  for  incomplete  design  efforts  and  lost  control  over  the 
design  process. 


DISCUSSION  OF  DETAILS 


Background 

An  acquisition  strategy  is  prepared  for  each  major  Defense 
acquisition  program.  The  acquisition  strategy  is  introduced 
during  the  concept  exploration/definition  phase  and  updated,  as 
necessary,  during  successive  phases  through  to  the  production 
phase.  The  acquisition  strategy  is  the  basic  planning  document 
for  each  major  Defense  acquisition  program.  An  acquisition  plan 
is  prepared  based  on  the  acquisition  strategy  and  identifies 
specific  contractual  actions  needed  to  develop  and  produce  the 
weapon  system  identified  in  the  acquisition  strategy.  Based  on 
the  need  to  clearly  and  precisely  communicate  Government 
requirements  and  intentions  to  industry,  requirements  necessary 
for  planning,  executing,  and  controlling  the  CDR  process  should 
be  addressed  in  the  acquisition  plan  and  included  in  requests  for 
proposals  made  to  contractors  for  the  engineering  and 
manufacturing  development  of  major  Defense  acquisition  programs. 

Planning  for  Critical  Design  Review 

Inadequate  Contract  Provisions.  Contracts  for  three  of 
the  six  programs  included  in  our  audit  lacked  provisions  to 
conduct  the  CDR  effectively.  As  the  following  table  shows, 
five  of  the  six  contracts  lacked  provisions  for  withholding 
payments  for  open  action  items  resulting  from  the  CDR  process: 
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Major 

Defense 

Acquisition 

Program 

Statement 
of  Work 

Contract 
Line  Item 
Number 

Contract 

Data 

Requirements 

List 

Withhold 

Provisions 

EPLRS 

No 

No 

Yes 

No 

Javelin 

Yes 

No 

Yes 

No 

V-22  Osprey 

Yes 

Yes 

Yes 

Yes 

SSN-21 

Yes 

Yes 

Yes 

No 

RSIP 

Yes 

Yes 

Yes 

NO 

Joint  STARS 

Yes 

No 

Yes 

No 

In  our  opinion,  contract  terms  for  effectively  implementing  CDRs 
should  minimally  include: 

o  a  Statement  of  Work  asking  the  contractor  to  plan  for  and 
conduct  a  CDR, 

o  a  separately  priced  contract  line  item  tied  to  the 
successful  completion  of  the  CDR  from  which  withholds  can  be 
established,  and 

o  withhold  provisions  to  ensure  satisfactory  completion  of 
action  items  resulting  from  the  CDR. 

Mo  contract  requirements  for  critical  design  review.  A 

CDR-planning  disconnect  existed  between  the  EPLRS  development 
contract  provisions  and  the  acquisition  plan.  The  EPLRS 
acquisition  plan  for  the  final  phase  of  development  required 
using  quarterly  design  reviews,  special  design  reviews, 
production  and  readiness  reviews,  and  technical  interchange 
meetings  to  monitor  contractor  progress.  The  acquisition  plan 
also  set  up  a  program  milestone  schedule  requiring  a  system 
design  review,  preliminary  design  review,  and  a  CDR  no  later  than 
12  months  after  contract  award.  However,  these  design 
reviews  were  not  included  in  the  Statement  of  Work.  Instead,  the 
procurement  contracting  officer  included  a  Statement  of  Work 
tasking  the  contractor  to  conduct  a  detailed  design  review  to 
establish  a  design  freeze  on  the  hardware  and  software  baselines. 
This  language  is  similar  to  language  used  in  Statements  of  Work 
that  require  contractors  to  conduct  a  CDR.  However,  the  language 
neither  required  that  the  detailed  design  review  be  based  on 
objective  criteria,  such  as  Mil-Std  1521B,  nor  was  the  language 
supported  by  a  contract  line  item  number. 

The  detailed  design  review  was  never  identified  as  such  in  the 
contractor's  cost  performance  reports  or  in  the  minutes  of  in- 
process  reviews  conducted  during  and  after  that  period.  The 
contractor  had  permission  from  the  program  office  to  replace  the 
detailed  design  review  with  two  program  reviews  in  August  and 
December  1986,  respectively,  and  one  production  readiness  review 
in  December  1986.  The  program  reviews  and  the  production 
readiness  review  did  not  provide  the  necessary  coverage, 
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identification,  and  closure  of  action  items  to  satisfy  the  effort 
required  of  the  CDR  as  outlined  in  Mil-Std  15216. 

Inadequate  or  unclear  Statement  of  Work.  The  E-8A  air¬ 
craft  was  selected  as  the  original  platform  for  the  Joint  STARS 
program.  In  the  E-8A  aircraft  contract,  CDR  was  discussed  only 
in  one  sentence  in  the  Statement  of  Work.  As  a  result  of  this 
lack  of  contract  emphasis  in  the  planning  stage,  the  program 
office  needed  a  concentrated  effort  to  get  the  contractor  through 
the  CDR.  Our  audit  found  that  the  contractor  clearly  did  not 
understand  what  was  expected  by  the  Air  Force  to  conduct  a 
successful  CDR.  However,  the  program  office  improved  the  CDR 
process  when  it  needed  to  replace  the  E-8A  aircraft  with  the  E-8C 
aircraft  as  a  platform  for  the  Joint  STARS  program.  It  developed 
a  contract  that  incorporated  an  excellent  Statement  of  Work  for 
the  CDR. 


Mo  provisions  for  withholds.  Five  of  the  six  major 
Defense  acquisition  programs  reviewed  did  not  include  provisions 
in  their  engineering  and  manufacturing  development  contracts  for 
withholding  progress  payments  for  incomplete  CDR  actions.  The 
Federal  Acquisition  Regulation  and  the  DoD  Supplement  to  the 
Federal  Acquisition  Regulation  provide  that  Administrative 
Contracting  Officers  have  the  authority  to  withhold  all  or  a 
portion  of  progress  payments  when  compliance  with  contract 
provisions  are  incomplete.  However,  in  our  opinion,  these 
general  provisions  of  the  Federal  Acquisition  Regulation  and  the 
DoD  Supplement  to  the  Federal  Acquisition  Regulation  are  subject 
to  contractor  misunderstanding  and  dispute.  For  example,  we 
found  that  the  Joint  STARS  and  EPLRS  program  offices  did  not 
employ  withholds  even  though  CDR  or  other  design  changes  caused 
significant  schedule  slips.  Including  provision  for  withholds 
for  incomplete  CDR  actions  in  the  Statement  of  Work  in  Requests 
for  Proposals  and  contracts  would  clarify  for  contractors  the 
importance  the  Government  places  on  CDR,  what  the  Government 
considers  full  compliance,  and  the  monetary  penalties  for 
incomplete  compliance. 

Superseded  military  specifications.  The  November  1990 
engineering  and  manufacturing  development  contract  for  E-8C  air¬ 
craft  used  as  platforms  for  the  Joint  STARS  program  incorporated 
Mil-Std  152 1A,  "Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Programs,"  June  1,  1976,  as  the  CDR 
criterion.  However,  Mil-Std  1521A  had  been  superseded  by  Mil-Std 
1521B  on  June  4,  1985.  The  new  E-8C  contract  should  have 
included  the  Military  Standard  in  effect  when  the  contract  was 
awarded . 

Program  office  personnel  said  that  Mil-Std  152 IB  was  not  used 
because  the  E-8C  aircraft  replaced  earlier  selected  E-8A  air¬ 
craft,  and  the  program  manager  did  not  want  to  use  CDR 
requirements  that  were  not  in  effect  for  the  E-8A  aircraft.  The 
program  manager's  strategy  for  E-8C  technical  review  included 
only  differences  between  the  E-8A  aircraft  and  E-8C  aircraft,  not 
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all  configuration  items  and  computer  software  configuration  items 
for  the  E-8C  aircraft.  However,  Mil-Std  1521B  is  more 
comprehensive  as  to  documentation  to  be  reviewed  during  CDR. 

Mil-Std  1521B  also  includes  two  more  significant  reviews,  the 
software  specification  review  and  the  test  readiness  review.  The 
software  specification  review  is  conducted  before  the  preliminary 
design  reviews,  and  critical  design  reviews  are  conducted  during 
the  engineering  and  manufacturing  development  phase  to  review  the 
final  computer  software  configuration  item  requirements  and 
operational  concept.  The  test  readiness  review  comes  after  the 
software  CDR  to  ensure  that  the  contractor  is  prepared  for  formal 
computer  software  configuration  item  testing. 

Performance  and  Control  of  Critical  Design  Review 

Although  five  of  the  six  systems  reviewed  had  formal  CDRs,  they 
did  not  accomplish  their  primary  purpose.  Specifically,  the 
design  reviews  were  done  when  detailed  design  of  the  system  was 
incomplete,  action  items  resulting  from  the  CDR  were  unresolved, 
determinations  were  not  made  as  to  whether  corrective  action  was 
contractor  or  Government  responsibility,  and  action  items  were 
inadequately  tracked  to  assure  that  design  issues  were  resolved. 
The  reviews  were  ineffective  because  no  criteria  or  procedures 
guided  the  CDR  process.  Another  factor  was  that  program 
milestone  schedules  drove  the  reviews.  As  a  result,  major 
technical  or  performance  issues  or  both  remained  unresolved  after 
the  CDRs  were  considered  complete.  This  delayed  program 
development  and  transferred  any  cost  and  technical  risk  of 
redesign  to  the  Government.  Appendix  B  summarizes  the  plan  and 
accomplishment  of  design  reviews  for  the  major  Defense 
acquisition  programs  included  in  this  audit.  Also,  Appendix  B 
summarizes  the  impact  of  delays  in  design  reviews  on 
configuration  audits  and  initial  operating  capability  for  each 
acquisition  program. 

Premature  and  Schedule-Driven  Critical  Design  Reviews.  The 
scheduling  and  executing  of  CDRs  were  premature  and  schedule- 
rather  than  event-driven.  CDR  should  not  be  started  until  the 
system  design  is  stable,  since  it  is  at  this  point  in  the 
development  that  the  Government  is  preparing  the  system  for  final 
configuration  and  transition  into  production.  The  CDR  should  set 
design  stability  and  end  with  acceptance  of  the  design.  A 
thorough  CDR  is  the  Government's  last  chance  to  identify  design 
weaknesses  and  to  determine  if  the  proposed  design  meets  contract 
specifications  and  performance  requirements.  However,  our  audit 
found  that  the  Government  proceeded  with  CDRs  even  when  events 
that  should  precede  the  CDR  had  not  occurred. 

The  CDR  conference  is  run  by  the  contractor.  Normally, 
engineering  and  manufacturing  development  contracts  assign 
responsibility  for  the  agenda  and  minutes  of  the  CDR  conference 
to  the  contractor.  The  contractor  must  present  proposed 
production  design  to  the  Government  at  this  point.  The 
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Government  must  critically  review  the  detailed  design  to 
determine  if  it  meets  the  contract  specification  and  performance 
requirements . 

Detailed  drawings  presented  to  the  Government  at  the  CDR 
conference  were  incomplete.  As  a  result,  compliance  with  contract 
specifications  and  performance  requirements  could  not  be 
determined,  solutions  to  apparent  technical  and  performance 
issues  could  not  be  determined,  and  the  contractor  could  not 
ensure  that  the  design  was  stabilized. 

Premature  critical  design  review.  Four  of  six  programs 
reviewed  held  the  CDR  before  the  detail  design  was  substantially 
complete.  Therefore,  major  unresolved  technical  issues  were 
known  before  holding  the  CDR. 

This  was  the  case  for  the  Javelin  and  V-22  Osprey  programs.  The 
CDR  for  the  Javelin  program  was  accomplished  and  most  action 
items  were  closed  even  though  testing  disclosed  major  design 
deficiencies  in  safety  and  performance.  The  Joint  STARS  and 
EPLRS  programs  also  entered  CDR  prematurely.  In  both  cases,  the 
maturity  of  system  design  was  not  the  driving  factor  for 
scheduling  the  CDRs.  Rather,  their  CDRs  were  tied  to  specific 
dates  set  in  the  contracts  or  based  on  preestablished  fielding 
plans.  An  accelerated  development  strategy  was  used  in  both 
programs,  which  may  have  led  to  this  condition.  The  remaining 
technical  issues  for  the  systems  reviewed  were  so  substantial 
that  the  stability  of  the  design  was  questionable.  By  conducting 
the  CDR  when  the  design  was  unstable,  the  basic  goals  of  the  CDR 
process  were  unattainable.  Detail  design  for  configuration  items 
should  be  mostly  complete  before  the  CDR.  However,  the  status  of 
drawings  and  the  solution  of  technical  problems  were  incomplete 
when  the  CDR  was  held.  This  approach  had  significant  adverse 
affects  on  the  programs  in  terms  of  risk,  schedule  delays,  cost 
increases,  and  design  instability.  Specifically,  premature 
acceptance  of  CDR  transfers  risk  associated  with  the  design  to 
the  Government. 

For  example,  our  audit  made  the  following  findings: 

o  CDR  for  the  V-22  Osprey  program  was  held  when  only 
70  percent  of  the  detailed  drawings  were  available. 

o  CDR  participants  for  the  Javelin  program  were  not  given 
an  opportunity  to  review  detailed  drawings  or  study  the  detailed 
design  before  the  CDR  conference.  After  the  CDR  was  considered 
complete,  later  testing  revealed  the  need  for  major  redesign  of 
Javelin  components.  No  plans  existed  to  conduct  a  review  of  the 
redesigned  components. 

o  More  contracts  for  design  studies  were  funded  for  the 
V-22  Osprey  program  following  the  CDR. 
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o  Technical  and  performance  issues  for  the  Javelin  program 
were  not  raised  in  the  CDR  conference  but  during  working  group 
committee  meetings  that  were  not  attended  by  all  participants. 

o  The  Net  Control  Station  Trainer,  developed  under  the 
EPLRS  program,  was  subject  to  a  CDR  by  Mil-Std  152 1A.  It  was 
identified  with  a  contract  line  item  number  for  $1.4  million  and 
delineated  in  the  Statement  of  Work.  The  award  to  manufacture 
the  Net  Control  Station  Trainer  was  based  on  the  Army's  need  to 
train  enough  operation  and  maintenance  personnel  for  the  Net 
Control  Station  by  the  f irst-unit-equipped  date.  Because 
deficiencies  were  found  during  technical  tests,  ending  in  March 
1989,  the  EPLRS  first-unit-equipped  date  went  from  April  1989  to 
a  conditional  first-unit-equipped  date  of  May  1994  and  an 
initial-operating-capability  date  of  February  1997.  As  a  result, 
the  production  Net  Control  Station  Trainer  delivered  in  1989  will 
need  to  be  reengineered  each  time  design  of  the  not-yet- 
stabilized  Net  Control  Station  is  changed. 

Schedule-driven  critical  design  review.  Our  review  of 
the  six  major  Defense  acquisition  programs  indicated  that  CDR 
emphasized  meeting  milestone  schedules  rather  than  the  readiness 
of  program  design  for  CDR.  For  example,  CDR  contract  provisions 
established  milestones  for  the  CDR  within  specified  months  of 
contract  award  rather  than  when  events,  such  as  completing 
detailed  drawings,  indicated  that  a  program  was  ready  for  CDR. 
In  the  JAVELIN  program,  an  initially  compressed  development 
schedule  arbitrarily  set  the  CDR  for  9  months  after  contract 
award . 

Proceeding  with  the  CDR  to  maintain  schedule  actually  prolonged 
the  process.  Government  decisions  to  accept  the  design  could  not 
be  made  because  the  CDR  was  premature  in  the  design  process. 
Holding  the  arbitrarily  scheduled  CDR  conference  did  not  result 
in  a  critical  review  of  the  design  but  rather  a  cursory  review  of 
drawings  presented  by  the  contractor.  The  CDR  should  result  in 
accepting  a  design  that  is  ready  for  manufacture.  This  was  not 
the  case  for  any  system  reviewed.  Extending  the  development 
phase  was  the  outcome.  The  very  thing  the  programs  hoped  to 
accomplish,  maintaining  schedule,  was  not  accomplished;  in  fact, 
it  was  extended. 

Even  though  the  EPLRS  program  did  not  conduct  a  formal  CDR,  the 
acquisition  strategy  was  based  on  fielding  EPLRS  a  year  before 
fielding  the  Forward  Area  Air  Defense  Command,  Control,  and 
Intelligence  system  for  which  EPLRS  was  to  provide  data 
communications.  This  was  a  primary  factor  for  following  contract 
schedule.  Accelerated  acquisition  to  include  concurrent 
development  and  testing  was  eliminated  from  the  EPLRS  program  in 
1989,  when  an  event-based  strategy  with  technical  baseline 
parameters  was  adopted. 
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The  Joint  STARS  program  established  ambitious  design  review 
milestones.  The  CDR  was  scheduled  for  19  months  after  the 
contract  was  signed.  However,  because  preliminary  design  review 
was  delayed,  the  CDR  was  held  38  months  after  the  contract  was 
signed.  The  System  Program  Director  delayed  preliminary  design 
review  because  the  contractor  was  not  ready  with  the  design  data 
and  solutions  needed  for  a  successful  preliminary  design  review. 
Even  after  the  preliminary  design  review,  the  lack  of  design 
maturity  became  apparent  during  CDR  when  the  level  of  design  data 
needed  for  successful  CDR  was  not  provided  and  agreement  could 
not  be  reached  on  how  to  close  open  action  items  from  both  the 
preliminary  and  critical  design  reviews. 

Conducting  design  reviews  when  programs  are  not  ready  increases 
the  risk  that  major  design  deficiencies  will  go  undetected  until 
later.  In  our  opinion,  the  CDR  for  the  Joint  STARS  program  was 
held  to  give  the  appearance  that  progress  was  being  made  and  that 
the  program  was  not  further  behind  the  preestablished  contract 
schedule  for  CDR.  Also,  it  appears  that  the  CDR  was  held  in  the 
expectation  that  needed  design  and  test  solutions  could  be  found 
in  the  follow-on  technical  reviews. 

Conducting  Critical  Design  Reviews.  The  CDR  conferences 
were  conducted  more  as  a  briefing  on  the  design  status  than  as  a 
CDR  to  determine  if  it  met  contract  and  performance  requirements. 
For  example,  we  found  no  challenges  to  the  system  design  by 
personnel  who  participated  in  the  CDRs.  Technical  issues 
affecting  overall  system  performance  were  deferred  to  working 
group  committees.  Usually,  regular  reviews  and  discussions  of 
system  design  take  place  between  the  system  program  office  and 
the  contractor,  while  working  groups  focus  on  specific  design 
problems.  These  meetings  must  not  replace  the  formal  CDR,  which 
should  lead  to  Government  acceptance  of  the  design.  The  deferral 
of  action  items  to  working  groups  does  not  complete  the  CDR  of 
the  component  or  system.  Instead,  the  working  group  sessions 
should  precede  the  CDR,  with  challenges  to  the  design  presented 
before  or  during  the  CDR.  By  not  adhering  to  essential  criteria 
or  establishing  guidelines  for  performing  the  CDR,  the  chance  for 
finding  technical  and  design  deficiencies  after  the  CDR 
increases,  which  delays  completing  the  program  and  increases 
program  cost. 

The  Navy's  V-22  Osprey  program  shows  how  conducting  the  CDR 
conference  contributes  to  ineffective  design.  The  V-22  Osprey 
program  office  accepted  the  contractor's  proposed  design  at  CDR, 
based  mainly  on  a  design-status  briefing  by  the  contractor.  To 
address  and  overcome  design  ambiguities,  the  program  office  later 
awarded  more  contracts  for  design  studies. 

Guidance  Provided  Attendees.  Existing  guidance  on  the 
CDR  process  does  not  set  forth  actions  expected  of  Government  and 
contractor  representatives.  Minutes  of  CDR  meetings  indicate 
that  more  than  100  people  attended  these  meetings.  However, 
notices  of  the  upcoming  CDRs  did  not  contain  information  or 
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instructions  to  attendees  as  to  their  responsibilities  during  the 
CDRs.  Most  attendees  had  no  input  to  the  CDR  meetings.  Again, 
the  CDR  seemed  to  be  a  cursory  review  of  the  design  presented  by 
the  contractor. 

control  of  Action  Items.  Two  of  the  six  major  Defense 
acquisition  programs  reviewed  inadequately  tracked  action  items 
generated  during  CDRs.  The  Federal  Acquisition  Regulation  and 
its  supplements  do  not  require  CDRs.  Contract  provisions  must 
establish  and  delineate  requirements  for  CDRs.  So,  the  contract 
provisions  for  CDR  determine  if  corrective  actions  for  action 
items  resulting  from  CDRs  are  within  the  scope  of  the  contract 
("in  scope"  items)  and  a  contractor's  responsibility  or  not 
within  the  scope  of  the  contract  ("out  of  scope"  items)  and  thus 
the  Government's  responsibility.  In  scope/out  of  scope  decisions 
should  be  made  when  action  items  resulting  from  CDR  are 
identified.  These  decisions  should  specify  if  the  required 
corrective  action  is  within  the  scope  of  the  contract  and  payable 
by  the  contractor  or  outside  the  scope  of  the  contract  and 
payable  by  the  Government. 

The  EPLRS  and  Joint  STARS  programs  did  not  track  or  maintain  the 
status  of  corrective  actions  for  all  action  items.  These  program 
management  offices  relied  on  their  contractors  to  maintain  such 
records.  We  got  the  documentation  needed  to  accomplish  our 
review  of  these  systems  from  contractors.  In  our  opinion,  to 
control  action  items  and  properly  ensure  adequate  resolution,  the 
program  office  must  maintain: 

o  complete  records  for  all  design  issues  requiring  action, 

o  determinations  whether  each  issue  is  in  scope  or  out  of 
scope, 

o  the  action  approved  to  resolve  each  issue,  and 

o  the  due  date  for  completing  the  approved  action. 

Managing  Critical  Design  Reviews.  All  major  Defense 
acquisition  programs  reviewed  had  shortcomings  in  CDR  management. 
Appendix  A  lists  each  program's  identified  shortcomings.  The 
Javelin  program  is  discussed  because  it  shows  many  of  the 
identified  shortcomings.  Most  critical  design  analyses  of  the 
Javelin  were  done  in  working  group  meetings  after  the  CDR  was 
considered  complete.  As  of  the  completion  of  our  Javelin  audit 
verification  work  in  May  1992,  the  program  was  still  trying  to 
resolve  design-related  technical  deficiencies  discovered  after 
the  CDR. 

Detailed  drawings  were  unavailable  before  the  CDR  to  determine 
compliance  with  performance  specifications.  Issues  reviewed  and 
discussed  during  the  CDR  were  those  found  in  the  normal  course  of 
a  development  program  and  were  easily  resolved.  The  critical 
review  of  proposed  design,  which  should  be  done  during  the  CDR, 
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was  done  instead  during  working  group  meetings  after  the  CDR. 
Also,  no  plans  exist  to  do  another  CDR  for  the  Javelin  program, 
although  at  least  two  major  components  will  be  redesigned. 

Program  management  representatives  conceded  that  the  CDR  was  held 
because  the  contract  scheduled  it  by  a  certain  date.  Although 
the  Javelin  program  was  not  ready  for  the  CDR  by  the  scheduled 
date,  the  CDR  was  conducted. 

For  the  EPLRS  program,  informal  technical  meetings  were  held 
instead  of  a  formal  CDR.  Although  the  EPLRS  system  is  an 
improvement  over  an  already  fielded  Position  Location  Reporting 
System,  the  importance  of  a  CDR  for  the  improvements  is  not 
lessened.  The  EPLRS  design  was  not  ready  when  the  CDR  was 
conducted  on  a  trainer  version  of  the  system.  Planned  design 
changes  for  the  system  were  postponed  as  growth  items.  Design 
was  insufficient  to  enable  its  acceptance.  A  properly  conducted 
CDR  should  result  in  Government  acceptance  of  the  design. 

Acceptance  of  Critical  Design  Reviews 

Mil-Std  152 IB  states  that  the  program  manager  provides  formal 
acknowledgment  of  completing  the  CDR  to  the  contractor  after 
receiving  the  CDR  minutes.  The  contracting  agency  sets  the 
adequacy  (acceptance)  of  the  contractor's  CDR  performance  by 
notification  of: 

o  approval  that  the  review  was  satisfactorily  completed, 

o  contingent  approval  that  the  review  is  not  considered 
completed  until  resolution  of  resultant  action  items,  or 

o  disapproval  if  the  review  was  seriously  inadequate. 

Acceptance  should  acknowledge  that  the  CDR  was  conducted  and 
resultant  action  items  have  been  resolved  to  the  Government's 
satisfaction. 

Process  Closed  Before  Action  Items  Complete.  The  program 
managers  and  procuring  contracting  officers  accepted  CDRs  with 
significant  action  items  unresolved  and  without  commensurate 
monetary  withholds  or  effective  tracking  to  ensure  satisfactory 
completion. 


Critical  design  review  as  a  contract  line  item.  When 
CDR  was  a  contract  line  item  and  withholds  were  exercised,  this 
control  operated  well  to  ensure  that  the  contractor  completed  the 
action  items  before  receiving  full  compensation  for  CDRs.  For 
three  of  the  five  programs,  their  contracts  included  a  contract 
line  item  with  a  separate  price  for  the  CDR.  This  contract 
leverage  gave  the  contracting  officers  control  of  the  action  item 
closure.  For  example,  though  the  contract  included  no  specific 
provision  for  withholding  payments  for  open  action  items 
resulting  from  the  CDR  process,  the  RSIP  program  office  continued 
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to  withhold  $851,922  until  all  action  items  were  successfully 
closed  out.  However,  unless  contract  leverage  is  enforced, 
accepting  CDRs  can  occur  with  action  items  open  after  the 
Government  has  paid  for  design  reviews.  This  situation  seriously 
compounds  the  difficulty  in  ensuring  satisfactory  closure  of  CDR. 

For  example,  the  V-22  Osprey  program  office  had  23  CDR  action 
items  open  at  the  completion  of  the  CDR  in  December  1986  and 
recommended  a  withhold  of  $4.5  million  to  encourage  quick  closure 
of  the  action  items.  However,  the  contractor  disagreed  with  the 
recommended  withhold  amount.  So,  the  withhold  was  reduced  to 
$500,000.  However,  even  this  reduced  withhold  amount  was  later 
released  to  the  contractor  while  action  items  had  not  been 
completed.  The  contracting  officer  during  our  audit  could  not 
explain  why  the  previous  contracting  officer  had  released  the 
entire  amount  withheld. 

Critical  design  review  when  not  a  contract  line  item. 
When  CDR  was  not  a  contract  line  item,  we  found  that  the  contract 
clauses  encouraged  accepting  CDR  and  deferring  completion  of  open 
action  items.  The  Javelin  and  Joint  STARS  programs  accepted 
their  CDRs  with  significant  unresolved  action  items.  Joint  STARS 
contract  clauses  defined  successful  completion  of  the  CDR  as 
’•full  compliance  with  Military  Standard  1521A,  and  publication  of 
critical  design  review  minutes  which  include  a  mutually  agreed  to 
schedule  for  completion  of  all  unresolved  tasks"  (emphasis 
added)  .  In  July  1990,  an  agreement  was  made  on  a  schedule  to 
close  out  35  of  the  remaining  56  action  items.  Moreover,  the 
Joint  STARS  contract  had  an  award  fee  tied  to  completing  the  CDR 
successfully. 

In  October  1990,  the  award  fee  board  approved  and  awarded 
$450,000  to  the  contractor  ($750,000  total  award  fee  available). 
The  contracting  officer  recommended  against  any  fee  because  the 
contractor's  performance  was  unsatisfactory  during  the  CDR.  In 
the  agreement,  the  contractor  was  reminded  that  although  the 
mutually  agreed  to  schedule  for  completing  unresolved  tasks 
signifies  the  closure  of  CDR  according  to  the  "special  contract 
requirements  section,"  the  contractor  is  still  responsible  for 
closing  out  open  items.  However,  the  Joint  STARS  program's 
release  of  the  award  fee  removed  the  incentive  for  quick  closure 
of  open  action  items.  The  contract  clauses  should  have  allowed  a 
monetary  withhold  commensurate  with  the  significance  of  the  open 
action  items  instead  of  rewarding  the  contractor  for 
unsatisfactory  performance. 

The  Government  lost  its  chance  to  encourage  contractors  through 
monetary  withholds  to  expedite  completion  of  the  open  action 
items.  As  a  result,  although  CDRs  were  closed,  production 
baselines  were  not  established  and  design  work  continued  on  three 
of  the  major  Defense  acquisition  programs  included  in  our  audit. 
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Reporting  Critical  Design  Review  Results  and  completion. 
Program  managers  needed  to  improve  their  reported  CDR  results  and 
completion  milestone  dates  in  the  "Defense  Acquisition  Executive 
Summary"  report  for  senior  acquisition  management's  use. 

DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies 
and  Procedures,"  February  23,  1991,  states  that  the  "Defense 
Acquisition  Executive  Summary"  report  should  provide  the 
milestone  decision  authority  with  adequate  information  to  oversee 
acquisition  programs  and  provide,  regularly  and  systematically, 
advance  indications  of  potential  and  actual  program  problems 
before  they  become  significant. 

Reported  results.  The  CDR  results  reported  to  senior 
acquisition  management  in  "Defense  Acquisition  Executive  Summary" 
reports  were  incomplete  and  inaccurate  because  CDR  has  not  been 
clearly  identified  as  a  reportable  baseline  requirement.  Our 
review  of  these  reports  for  five  of  the  six  major  Defense 
acquisition  programs  showed  that  the  reported  CDR  results  were 
incomplete  and  inaccurate.  The  program  managers  did  not  report 
CDR-r elated  events  sufficiently  to  track  the  CDRs  progress  from 
report  to  report.  For  each  of  the  five  acquisition  programs,  the 
information  reported  was  insufficient  to  determine  the  CDRs 
status  or  results.  For  example,  we  reviewed  the  reported  results 
in  nine  reports  for  the  Joint  STARS  program  from  starting  the  CDR 
through  resolving  the  open  action  items  (November  1988  to  March 
1992) .  We  found  only  two  comments  that  addressed  CDR  results. 

o  August  1989  report  date:  "Grumman  is  developing  the  plan 
to  resolve  remaining  open  critical  design  review  action  items." 

o  February  1989  report  date:  "The  software  test  portion  of 
critical  design  review  was  successfully  completed." 

The  reported  information  was  incomplete  and  misleading.  The  plan 
to  resolve  the  remaining  CDR  open  action  items  was  not  agreed  to 
until  July  1990,  1  year  after  the  reports  stated  that  the 
contractor  was  developing  the  plan.  Moreover,  although  the 
software  test  portion  of  the  CDR  was  reported  as  completed,  our 
review  showed  that  66  action  items  were  open.  So,  actually  it 
was  not  completed  and  could  not  be  completed  until  the  action 
items  were  closed. 

Such  reporting  is  misleading  and  does  not  give  accurate  CDR 
results.  This  condition  occurs,  in  part,  because  DoD  Instruction 
5000.2  does  not  address  how  to  report  CDR  results,  status,  and 
completion.  Also,  program  managers  were  not  always  thorough  in 
reporting  CDR  results.  So,  the  information  reported  in  the 
"Defense  Acquisition  Executive  Summary"  was  not  useful  to  the 
senior  acquisition  management. 
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Although  the  Javelin  and  Joint  STARS  programs  had  been 
restructured  by  the  Defense  Acquisition  Board  (in  September  1991 
and  October  1989,  respectively),  few  open  action  items  had  been 
closed  during  the  programs'  restructuring  periods.  So,  we  do  not 
believe  the  Defense  Acquisition  Board  was  fully  aware  of  the  open 
action  items;  we  question  how  lacking  such  knowledge  affected  the 
decisionmaking  process.  For  example,  we  believe  the  Conventional 
Systems  Committee  of  the  Defense  Acquisition  Board  approved  the 
advance  buy  of  components  for  the  Joint  STARS  program  without 
complete  knowledge  of  the  CDR  status,  the  status  of  action  items 
resulting  from  the  CDR,  an  assessment  of  the  impact  of  open 
action  items  on  the  program's  ability  to  meet  the  users  needs,  or 
the  estimated  completion  date  for  the  CDR. 

Documented  completion  dates.  The  acquisition  program 
baselines  did  not  include  CDR  completion  dates.  The  most- 
repeated  phrase  in  the  "Defense  Acquisition  Executive  Summary" 
reports  was  that  "the  critical  design  review  has  been 
successfully  completed."  However,  this  meant  only  that  the  CDR 
conference  was  held,  not  that  CDR  had  actually  been  completed. 
There  was  no  information  on  the  action  items  resulting  from  the 
CDR  conference,  on  whether  the  Government  and  contractor  had 
mutually  agreed-on  actions  to  close  out  the  items,  or  on  the 
mutually  agreed  proposed  completion  dates. 

Decisionmakers  in  the  acquisition  chain  can  effectively  control 
major  Defense  acquisition  programs  only  if  they  are  kept  informed 
of  emerging  problems.  The  information  needed  comes  from  a 
monitoring  system  based  on  the  premise  of  management  by 
exception.  Acquisition  program  baselines  are  an  integral  part  of 
the  DoD  monitoring  system.  DoD  Instruction  5000.2,  part  11, 
section  A,  requires  that  acquisition  program  baselines  contain 
objectives  for  key  cost,  schedule,  and  performance  parameters. 
Key  parameters  include  thresholds  beyond  which  the  program 
manager  may  not  trade  off  without  authorization  from  the  Defense 
Acquisition  Board.  DoD  Instruction  5000.2,  part  16,  section  F, 
lists  CDR  as  a  key  schedule  parameter,  thus  requiring 
notification  to,  and  authorization  from,  the  Defense  Acquisition 
Board  in  the  event  of  CDR  schedule  breach.  However,  DoD 
Instruction  5000.2  does  not  specify  if  the  baseline  schedule 
established  for  the  CDR  should  be  the  CDR  initiation,  the  CDR 
conference,  or  the  CDR  completion. 

Although  CDR  dates  were  included  in  baselines  for  the  five  major 
Defense  acquisition  programs  in  our  audit  that  had  CDR 
contractual  requirements,  the  programs  did  not  list  CDR 
completion  dates.  No  such  requirement  exists  in  DoD  Instruction 
5000.2,  although  such  information  may  be  provided. 

Though  not  required  because  its  CDR  baseline  included  only  the 
scheduled  CDR  start  date,  the  RSIP  program  office  provided  a 
deviation  report  to  senior  acquisition  management  in  August  1991 
when  the  system  CDR  was  delayed  by  8  months.  All  program  offices 
would  be  required  to  provide  deviation  reports  to  senior 
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acquisition  management  if  the  CDR  completion  dates  were  specified 
in  the  baseline  and  breached.  Program  managers  should  document 
CDR  completion  dates  in  their  baseline  schedules  so  that 
slippages  in  completion  dates  are  brought  to  the  attention  of 
senior  acquisition  management,  explained,  and  corrected  while  the 
Government  still  controls  the  design  process. 

Conclusion 

The  CDR  process  deficiencies  identified  in  our  audit  were  due  to 
inadequacies  in  and  ineffective  compliance  with  existing 
directions,  guidance,  and  procedures  for  planning,  performance, 
control,  and  acceptance  of  the  CDR  process.  For  example,  CDR 
completion  dates  were  not  included  in  acquisition  program 
baselines  because  DoD  Instruction  5000.2  does  not  clearly  specify 
such  a  requirement.  Because  CDR  completion  dates  are  not 
included  in  baselines,  no  requirement  exists  to  notify  or  obtain 
authorization  from  the  Defense  Acquisition  Board  when  scheduled 
completion  dates  are  breached.  As  a  result,  acquisition 
executives  perform  oversight  responsibilities  without  knowledge 
of  breaches  in  scheduled  CDR  completion  dates. 

When  CDR  was  a  contract  line  item  and  withholds  were  exercised, 
we  found  that  this  control  ensured  that  the  contractor  completed 
the  action  items  before  receiving  full  compensation  for  CDRs.  We 
feel  that  this  control  should  be  mandated  for  all  major  Defense 
acquisition  programs. 

We  found  that  CDRs  were  conducted  and  closed  prematurely  because 
of  adherence  to  schedules  established  in  development  contracts, 
rather  than  adherence  to  a  logical  sequence  of  demonstrated 
accomplishments.  Acquisition  strategies  should  be  event-driven 
and  based  on  rigorous,  objective  assessments  of  a  program's 
status  and  the  plans  for  managing  design  risk.  The  acquisition 
strategy  and  associated  contracting  activities  must  explicitly 
link  milestone  decision  reviews  to  accomplishments  in 
development,  testing,  and  initial  production.  Exit  criteria 
should  be  set  for  CDR  initiation  and  completion.  The  acquisition 
strategy  must  reflect  the  interrelationships  and  schedule  of 
acquisition  phases  and  events  based  on  a  logical  sequence  of 
demonstrated  accomplishments,  not  on  fiscal  or  calendar 
expediency. 

At  each  milestone  decision  point,  assessments  should  be  made  of 
the  status  of  program  execution  and  the  plans  for  the  program's 
remaining  phases.  The  risks  associated  with  the  program  and  the 
adequacy  of  risk  management  planning  must  be  explicitly 
addressed.  Also,  critical  program-specific  results  required  in 
each  phase,  called  exit  criteria,  should  be  set.  Exit  criteria 
should  be  gates  through  which  a  program  must  pass  during  that 
phase.  DoD  Instruction  5000.2,  part  2,  paragraph  B.3.b., 
provides  that  exit  criteria  can  include  the  requirement  to 
conduct  a  CDR  before  committing  funds  for  long  lead-item 
procurement. 
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Decisionmakers  in  the  acquisition  chain  can  effectively  control 
major  Defense  acquisition  programs  only  if  they  are  informed  of 
emerging  problems.  We  believe  that  the  CDR  results,  including 
changes  in  contract  specifications  and  performance  of  mission 
requirements,  should  be  reported  as  they  occur  to  the  chairperson 
of  the  appropriate  Defense  Acquisition  Board  committee  and  the 
Service  acquisition  executive  for  all  major  Defense  acquisition 
programs. 

We  also  identified  some  deficiencies  in  the  control  of  the  CDR 
process.  Impending  revision  of  Mil-Std  1521B,  or  equivalent 
guidance,  should  include  procedures  to  track  action  items, 
document  compliance  with  all  CDR  functions,  and  provide 
information  and  guidance  to  CDR  attendees. 

We  have  reviewed  aspects  of  the  Navy  Technical  Education  Program 
related  to  the  design  review  process  and  consider  the  material 
appropriate  for  training  and  instructing  participants  in  critical 
design  reviews.  The  conduct  of  a  design  review  in  accordance 
with  the  guidelines  established  in  the  Navy  program  would  assist 
in  correcting  certain  deficiencies  identified  in  this  report. 

RECOMMENDATIONS  FOR  CORRECTIVE  ACTION 
We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition: 

1.  Require  for  all  major  Defense  acquisition  programs  that 
completion  of  critical  design  review  be  made  an  acquisition 
program  baseline  as  defined  by  DoD  Instruction  5000.2,  "Defense 
Acquisition  Management  Policies  and  Procedures,"  February  23, 
1991,  with  a  scheduled  completion  date.  Only  the  Under  Secretary 
of  Defense  for  Acquisition  should  be  able  to  waive  this 
requirement. 

2.  Establish  exit  criteria  in  the  integrated  program 
summaries  (supporting  milestones  I  and  II)  for  starting  and 
completing  all  phases  of  initial,  preliminary,  and  critical 
design  review. 

3.  Require  that  completing  critical  design  review  (defined  as  a 
mutually  agreed  plan  of  action,  approved  by  the  Government,  with 
in  scope /out  of  scope  determinations  resolved)  be  an  exit 
criterion  for  proceeding  with  entry  into  production  (prototype, 
pilot,  or  low-rate  production) . 

4 .  Make  the  critical  design  review  a  separately  priced 
contract  line  item  in  development  contracts  with  contractual 
provisions  for  individual  withholds  for  each  open  action  item 
until  action  items  are  closed  out. 
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5.  Require  reporting  of  the  actual  results  of  critical 
design  reviews,  including  changes  in  contract  specifications  and 
performance  on  mission  requirements,  to  the  chairperson  of  the 
cognizant  Defense  Acquisition  Board  committee  and  Service 
acquisition  executive  for  all  major  Defense  acquisition  programs. 

6.  Expedite  issuance  of  the  revised  Military  Standard  1521B, 
••Technical  Reviews  and  Audits  for  Systems,  Equipments,  and 
Computer  Software,"  June  4,  1985,  or  equivalent  guidance,  to 
include  specific  procedures  for: 

o  tracking  actions  to  close  out  open  action  items; 

o  documenting  compliance  with  all  critical  design  review 
functions  (such  as  recording  minutes  of  critical  design  review 
meetings,  attendees,  agreed-to  actions) ;  and 

o  providing  adequate  data,  guidance,  and  instructions  to 
critical  design  review  attendees  well  before  critical  design 
review  meetings. 

MANAGEMENT  COMMENTS 

We  requested  that  comments  on  the  draft  report  be  provided  to  us 
by  August  31,  1992.  As  of  October  28,  1992,  we  had  not  received 
responses  to  the  draft  report.  However,  we  were  informed  that 
comments  were  being  prepared  and  would  be  submitted  in  response 
to  this  final  report. 
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APPENDIX  A  -  SUMMARY  Of  DEFICIENCIES  FOR  EACH  MAJOR  DEFENSE  ACQUISITION  PROGRAM  AUDITED 
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Lack  of  Senior  Management  Review 
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APPENDIX  B  •  PUN  ANC  ACCOMPLISHMENT  OF  CRITICAL  DESIGN  REVIEW  FOB  PROGRAMS  AUDITED 


CO  Cfl 


5  ~  2 

— *  4J  <  O 

SOU  *-> 

o  u  a.  w 

•*-  3  v  ft) 

SL  *,  - 

XI  H-  >r-  X 

a  g  3 

u  < 


0)  CL 

mm* 

<0 

•  *■>  p 

to  (A  9 


ui  ui  tu  uj  ui 

g  g  g  g  g 


§«*S  §  g  3  5  § 

o  8  rvj  nT  ™  c>  ^ 


St*  8  §  §  8  O 

— »  O  "s.  's.  "*x  *>**. 

cl  o  co  kv  r\j  *- 


c>  fe 


gj  I  g  g  I  I  I 


SI  s 


ru  «- 


CD  4-*  ft)  Of  U> 

m  t-  to  wo  s  ‘"'» 

«  *->  T3  CO  ft)  Of 


CJ  2  = 


UJ  UJ 

E  i 


UJ  UJ 

g  g 


^0.0 
f\J  x  crt 
*  ui  o 
O  £ 

^  4-»  Q_ 

8* 


i  8-  s 

♦*-  u  re 


o 

<0 

CO 

cn 

« 

-J 

c 

3 

i 

ft) 

z 

> 

L. 

iZ 

ft) 

Of 

ft) 

CL 

to  mJ 

I  g 


25 


Replaced  by  Detailed  Design  Review 
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APPENDIX  C  -  PRIOR  AUDITS  AND  OTHER  REVIEWS 


DoD  Inspector  General  Audits 

o  Report  92-112,  "The  Ultra-High  Frequency  Follow- 
on  Satellite,"  June  30,  1992,  disclosed  that  CDR  was  not 
planned  until  after  complex  components  of  the  design  were 
fabricated  and  assembled.  This  condition  existed  because 
the  program  office  did  not  implement  the  provisions  in 
Military  Standard  152 IB,  "Technical  Reviews  and  Audits 
for  Systems,  Equipments,  and  Computer  Software," 
December  19,  1985,  which  encourages  the  performance  of 
incremental  CDRs  during  weapon-system  development.  As  a 
result,  the  design  deficiencies  identified  at  the  planned 
CDR  could  adversely  affect  satellite  number  4  deployment 
requirements  or  result  in  the  Government  paying  the 
contractor  90  percent  of  the  costs  for  a  dysfunctional 
satellite  extremely  high  frequency  capability. 

o  Report  91-007,  "Selected  Acquisition  Actions  in  the 
C-17  Aircraft,"  November  2,  1990,  reported  that  a 

contract  option  for  four  aircraft  was  exercised  before 
completing  CDR  of  all  mission  computer  software. 
Completion  of  the  CDR,  a  prerequisite  to  the  option,  was 
required  by  a  November  1988  modification  of  the  full- 
scale  development  contract.  The  C-17  program  director 
considered  the  CDR,  made  in  April  1989,  to  be  adequate 
and  complete,  although  the  contractor  had  completed  the 
detail  design  for  only  about  60  percent  of  all  mission 
computer  software.  The  C-17  program  development  had 

experienced  major  delays.  Unless  all  required  mission 
computer  software  was  developed  before  the  C-17  passed 
from  the  development  test  and  evaluation  phase  to  the 
operational  test  and  evaluation  phase,  the  C-17  program 
schedule  would  be  further  delayed. 

o  Report  89-077,  "Acquisition  of  the  V-22  Joint  Services 
Advanced  Vertical  Lift  Aircraft,"  June  14,  1989,  reported 
that  the  portions  of  the  CDR  completed  at  the  time  of  the 
audit  followed  applicable  military  standards  and 

specifications.  The  Naval  Air  Systems  Command  had 

identified  14  issues  that  were  not  to  be  closed  until 
"proof  of  concept"  occurred  during  the  first  flight. 
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APPENDIX  C  -  PRIOR  AUDITS  AND  OTHER  REVIEWS  (Continued) 


General  Accounting  Office  Audits 

o  Report  IMTEC-91-30,  "Submarine  Combat  System,"  August 
1991,  reported  that  the  Navy  and  contractor  considered 
CDR  complete  although  required  design  tasks  were  still 
ongoing.  Navy  and  contractor  officials  stated  that  CDR 
was  completed  in  September  1990;  then  the  contractor 
began  coding  and  hardware  development.  However,  certain 
detailed  design  tasks,  contained  in  Defense  Standard 
2167,  "Defense  System  Software  Development,"  June  4, 
1985,  and  incorporated  into  the  contract,  were  to  be 
completed  before  CDR  but  were  still  ongoing  in  November 
1990.  For  example,  detailed  design  descriptions  for  the 
software  that  will  control  the  Seawolf's  weapons  or  the 
data  base  management  system  that  will  store  critical 
navigation  and  target  data  used  for  launching  these 
weapons  were  not  completed  in  September  1990.  These 
descriptions  are  critical  because  they  provide  a 
blueprint  for  programmers  to  code  the  system.  Coding 
done  before  completing  these  tasks  will  have  to  be 
modified  if  the  code  does  not  conform  to  the  eventual 
detailed  design  description. 


Naw  Special  Administrative  Inquiry 

o  Naval  General  Counsel  Report,  "A-12  Administrative 
Inquiry,"  November  29,  1990,  found  that  the  Navy  accepted 
CDR  as  complete  although  many  significant  issues  were 
unresolved.  Associated  testing  requirements  for  CDR 
Phase  II  were  also  accepted  as  complete  although  an 
aircraft  mock-up  meeting  contract  specifications  was  not 
available  for  required  critical  testing. 
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APPENDIX  D  -  SUMMARY  OF  POTENTIAL  BENEFITS  RESULTING  FROM  AUDIT 


Recommendation 


Reference 

Description  of  Benefit 

1. 

Internal  control. 

Require  that  completing 

CDR  be  made  an 
acquisition  program 
baseline  with  a 
scheduled  completion 
date. 

Undeterminable 
monetary  benefit. 
Advantages  of 
improved  control 
over  the  CDR 
are  not  readily 
quantifiable. 

2. 

Internal  control. 

Establish  exit 
criteria  for  starting 
and  completing  CDR. 

Undeterminable 
monetary  benefit. 
Affects  of 
delaying  CDR 
until  the  program 
is  adequately 
prepared  are  not 
readily 
quantifiable. 

3. 

Internal  control. 

Require  that  completing 

CDR  be  made  an  exit 
criterion  for  proceeding 
with  production 
(prototype,  pilot,  or 
low-rate  production) . 

Undeterminable 
monetary  benefit. 
Benefits  of 
delaying 
production 
until  all  design 
issues  are 
adequately 
resolved  are  not 
readily 
quantifiable. 

4. 

Internal  control. 

Make  CDR  a  separately 
priced  contract 
line  item  with 
provisions  for 
individual  withholds 
for  open  action 
items. 

Undeterminable 
monetary  benefit. 
Advantages  of 
improved  control 
over  the  CDR 
process  are  not 
readily 
quantifiable. 
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APPENDIX  D  -  SUMMARY  OF  POTENTIAL  BENEFITS  RESULTING  FROM  AUDIT 
(Continued) 


Recommendation 

Reference  Description  of  Benefit 


Type  of  Benefit 


5. 


Internal  control. 
Report  the  actual 
CDR  results  to  the 
Defense  Acquisition 
Board  and  Service 
acquisition  executive. 


Undeterminable 
monetary  benefit. 
Affects  of 
providing  senior 
acquisition 
management  the 
data  needed  to 
exercise  program 
oversight  are  not 
readily 
quantifiable. 


6. 


Internal  control. 
Expedite  issuance  of 
procedures  for: 

o  tracking  action 
items; 

o  documenting 

compliance  with 
all  CDR  functions; 
and 

o  providing  data, 
guidance,  and 
instructions  to 
attendees  well 
before  CDR 
meetings. 


Undeterminable 
monetary  benefit. 
Advantages  of 
improved  control 
of  the  CDR 
process  are  not 
readily 
quantifiable. 
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APPENDIX  E  -  ACTIVITIES  VISITED  OR  CONTACTED 


Office  of  the  Secretary  Of  Defense 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Washington,  DC 


Department  of  the  Army 

Office  of  the  Assistant  Secretary  of  the  Army  (Research, 
Development  and  Acquisition) ,  Washington,  DC 
All  Source  Analysis  System  Program  Office,  McLean,  VA 
EPLRS  Program  Office,  Fort  Monmouth,  NJ 
Javelin  Program  Office,  Redstone  Arsenal,  AL 


Department  of  the  Navy 

Office  of  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition) ,  Washington,  DC 
SSN-21  Seawolf  Program  Office,  Naval  Sea  Systems  Command, 
Washington,  DC 

V-22  Osprey  Program  Office,  Naval  Air  Systems  Command, 
Washington,  DC 


Department  of  the  Air  Force 

Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) , 
Washington,  DC 

Air  Force  Program  Executive  Office  Organization,  Washington,  DC 
AWACS  RSIP  System  Program  Office,  Electronics  Systems  Division, 
Hanscom  Air  Force  Base,  MA 

Joint  STARS  System  Program  office,  Electronics  Systems  Division, 
Hanscom  Air  Force  Base,  MA 

Peacekeeper  Rail  Garrison  System  Program  Office,  Ballistic 
Missile  Organization,  Norton  Air  Force  Base,  CA 
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APPENDIX  E  -  ACTIVITIES  VISITED  OR  CONTACTED  (Continued) 


Defense  Agencies 

Defense  Logistics  Agency,  Cameron  Station,  Alexandria,  VA 

Defense  Plant  Representative  Office,  Bell  Helicopter 
Corporation,  Fort  Worth,  TX 

Defense  Plant  Representative  Office,  Boeing  Aerospace  Company, 
Seattle,  WA 

Defense  Plant  Representative  Office,  Boeing  Vertol  Company, 
Philadelphia,  PA 

Defense  Plant  Representative  Office,  Grumman  Melbourne  Systems 
Division,  Melbourne,  FL 

Defense  Plant  Representative  Office,  Hughes  Aircraft 
Corporation,  Fullerton,  CA 

Defense  Plant  Representative  Office,  Texas  Instrument 
Corporation,  Dallas,  TX 

Defense  Plant  Representative  Office,  Westinghouse  Electric 
Corporation,  Baltimore,  MD 


Non-Government  Activities 


Bell  Helicopter  Corporation,  Fort  Worth,  TX 
Boeing  Aerospace  Company,  Seattle,  WA 
Boeing  Vertol  Company,  Philadelphia,  PA 
Grumman  Melbourne  Systems  Division,  Melbourne,  FL 
Hughes  Aircraft  Corporation,  Fullerton,  CA 
Texas  Instrument  Corporation,  Dallas,  TX 
Westinghouse  Electric  Corporation,  Baltimore,  MD 


32 


APPENDIX  F  -  REPORT  DISTRIBUTION 


Office  of  the  Secretary  of  Defense 
Under  Secretary  of  Defense  for  Acquisition 


Department  of  the  Army 
Secretary  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and 
Acquisition) 

Inspector  General,  Department  of  the  Army 
Program  Manager,  All  Source  Analysis  System 
Program  Manager,  Enhanced  Position  Location  Reporting  System 
Program  Manager,  Javelin  (Advanced  Antitank  Weapon  System) 


Department  of  the  Naw 
Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 
Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition) 

Program  Manager,  SSN-21  Seawolf 
Program  Manager,  V-22  Osprey 


Department  of  the  Air  Force 
Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and 
Comptroller) 

System  Program  Office,  Joint  Surveillance  Target  and  Attack 
Radar  System 

System  Program  Office,  Peacekeeper  Rail  Garrison 
System  Program  Office,  Radar  System  Improvement  Program  for  the 
Airborne  Warning  and  Control  System 


Defense  Aaencv 
Defense  Logistics  Agency 
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APPENDIX  F  -  REPORT  DISTRIBUTION  (Continued) 


Non-DoD  Federal  Organizations 
Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and 

International  Affairs  Division,  Technical  Information  Center 

Congressional  Committees: 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 

Ranking  Minority  Member,  Senate  Committee  on  Armed  Services 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
Ranking  Minority  Member,  House  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 
House  Subcommittee  on  Legislation  and  National  Security, 
Committee  on  Government  Operations 
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